2 Amazon Simple Storage Service (AWS S3)

2.1 Specific configuration for S3

For example, you can define a default bucket for file uploads. Everytime you attemp to upload a file without explicitly setting the bucket name, this default will be used.

Check now, all default config possibilities you can set.

Bucket name

To set a default bucket, that will be used to all file upload use the config below

grails {
   plugin {
      aws {
         s3 {
            bucket = "grails-plugin-test"
         }
      }
   }
}

During the first upload on this bucket, it will be created if does not exist.

ACL (file permission)

The permissions that will be granted on this file, you can use:

• public: Allow public access to everyone that attemps to read this file
• private: Sets private access to this file, only your account will read/write on it
• public_read_write: This will make this file wide open to any AWS account, read and write
• authenticated_read: Using this acl string, only logged AWS accounts will have permissions to read the file

To configure public access as default to all file uploads, use this:

grails {
   plugin {
      aws {
         s3 {
            acl = "public"
         }
      }
   }
}

If you like to set private access to your files, you should config this way:

grails {
   plugin {
      aws {
         s3 {
            acl = "private"
         }
      }
   }
}

RRS - Reduced Redundancy Storage

RRS stored files provides a cheaper storage with 99.99% durability instead of 99.999999999% as the default provided by AWS S3. More information here: http://aws.amazon.com/about-aws/whats-new/2010/05/19/announcing-amazon-s3-reduced-redundancy-storage/

This is disabled by default, if you like to set RRS enabled for all uploads, use this config key:

grails {
   plugin {
      aws {
         s3 {
            rrs = true
         }
      }
   }
}

2.2 Uploading files

Simple File upload from a File object

def s3file = new File("/tmp/test.txt").s3upload {
    path "folder/to/my/file/"
}

This way your test.txt file will be uploaded to

<default-bucket>.s3.amazonaws.com/folder/to/my/file/test.txt

If you want to overwrite config during the file upload, check this guide next section.

Uploading files directly from its InputStream

This is useful when you don't have the file stored in your filesystem. When user uploads files to your application using a multipart/form-data form, you can upload it directly to s3. Imagine it have an upload form like this:

<g:uploadForm action="uploadFromInputStream">
    <input type="file" name="photo">
    <input type="submit" value="upload">
</g:uploadForm>

you could have your uploadFromInputStream action implemented this way:

def file = request.getFile('photo')
def uploadedFile = file.inputStream.s3upload(file.originalFilename) {
    bucket "file-upload-from-inputstream"
}

Uploading from InputStream requires one extra parameters, the filename that this object will have in S3, usually the 'originalFileName'.

Note that when you use File.s3upload you just pass the closure that configures it. When using from one inputStream, you SHOULD have to specify the name that file will have and the file size. The above example show exactly how to do it with the correct info from the uploaded file.

Storing uploaded information for later use

If you are uploading some picture to S3, you'll probably need to store information on how to get that file again later.

The s3upload operation returns an instance of grails.plugin.aws.s3.S3File. As this plugin uses jets3t (http://jets3t.s3.amazonaws.com/index.html) to handle file upload, the S3File is just a wrapper for a delegated jets3t S3Object instance as you can see below:

package grails.plugin.aws.s3
import org.jets3t.service.model.S3Object
class S3File {
    @Delegate S3Object source
    public S3File(S3Object _source) {
        this.source = _source
    }    
}

So, you can call any S3Object method on S3File instance. S3Object API is available here: http://jets3t.s3.amazonaws.com/api/org/jets3t/service/model/S3Object.html, for example, to retrieve the ETag hash for the S3File uploaded you would just call:

def s3file = … //upload the file
def etag = s3file.getETag()

The S3File object returned will give all information you'll need. Now, depends on what information you want to store. Follow S3Object docs (link above) and get whatever you want.

A common approach would be storing the bucket, path and file (key). With these parameters you can rebuild the URL to reach the file, or even delete the file.

2.2.1 Setting file virual path

The effect is exactly like in a regular folder. For example, doing the upload below:

def uploadedFile = new File("/tmp/profile-picture.jpg").s3upload {
    bucket "my-aws-app"
    path "pictures/user/profile/"
}

The file will be stored and available in the following url:

http://my-aws-app.s3.amazonaws.com/pictures/user/profile/profile-picture.jpg

And using the AWS S3 console, the files will visually be inside folders either. Some third-party apps is already using this feature to show "folders".

2.2.2 Overwriting AWS credentials

def uploadedFile = new File("/tmp/test.txt").s3upload {
    credentials "my-other-access-key", "my-other-secret-key"
}

2.2.3 Overwriting bucket to file upload

def uploadedFile = new File("/tmp/test.txt").s3upload {
    bucket "other-bucket"
}

This file will be uploaded to

other-bucket.s3.amazonaws.com/test.txt

Remember, when plugin created a non pre-existent bucket, it will be created in the default US region. If you like to set a different location, just pass a second string parameter containing the region string. For example, to set this bucket creation in Europe region:

def uploadedFile = new File("/tmp/test.txt").s3upload {
    bucket "bucket-not-yet-created-in-europe", "EU"
}

2.2.4 ACL (file permission)

def uploadedFile = new File("/tmp/test.txt").s3upload {
    acl "private"
}

2.2.5 RRS - Reduced Redundancy Storage

def uploadedFile = new File("/tmp/test.txt").s3upload {
    rrs false
}

2.2.6 Setting File Metadata

def uploadedFile = new File("/tmp/test.txt").s3upload {
    metadata [user-id: 123, username: 'johndoe', registered-date: new Date().format('dd/MM/yyyy')]
}

2.2.7 Creating public URLs for private files

For example:

def s3file = new File("test.txt").s3upload {
   bucket "secret-files"
   acl "private"
}

This will make the file above (http://secret-files.s3.amazonaws.com/test.txt) inaccessible using its URL. If you need to allow someone to get the file in a short period of time, you can ask the plugin to generate a public URL for it. The object will remain private, but you can use the returned URL to access the file in a short period of time. For example:

def s3file = new File("test.txt").s3upload {
   bucket "secret-files"
   acl "private"
}
def publicUrl = s3file.publicUrlFor()

So, the value on the publicUrl is the URL for accessing the file. By default the URL will be valid for 1 hour. Every requests on it will return the uploaded object. After one hour, next requests on it will return an error: "Access Denied for Object".

Defining when public URL will expire

You can set the expires date for the public URL, passing an argument to publicUrlFor method.

s3file.publicUrlFor(3.hours) //will be available for 3 hours
s3file.publicUrlFor(10.years) //available for 10 years
s3file.publicUrlFor(1.second) //you won't get this one on time

You can any of these methods:

1.second or 2.seconds 
1.minute or 2.minutes
1.hour or 2.hours
1.day or 2.days
1.month or 2.months
1.year or 2.years

This properties are injected on Integer class, so, enjoy the magic.

2.2.8 Creating torrent for S3 hosted files

After uploading some file, just call the torrent() method on it and you'll get the torrent URL for it.

def s3file = new File("test.txt").s3upload {
   bucket "secret-files"
   acl "private"
}
def torrentUrl = s3file.torrent()

From the AWS docs:

There is no extra charge for use of BitTorrent with S3. Data transfer via the BitTorrent protocol is metered at the same rate as client/server delivery. To be precise, whenever a downloading BitTorrent client requests a “piece” of an object from the S3 “seeder”, charges accrue just as if an anonymous request for that piece had been made using the REST or SOAP protocol. These charges will appear on your S3 bill and usage reports in the same way. The difference is that if a lot of clients are requesting the same object simultaneously via BitTorrent, then the amount of data S3 must serve to satisfy those clients will be lower than with client/server delivery. This is because the BitTorrent clients are simultaneously uploading and downloading amongst themselves. The data transfer savings achieved from use of BitTorrent can vary widely depending on how popular your object is. Less popular objects require heavier use of the “seeder” to serve clients, and thus the difference between BitTorrent distribution costs and client/server distribution costs may be small for such objects. In particular, if only one client is ever downloading a particular object at a time, the cost of BitTorrent delivery will be the same as direct download.

2.3 Deleting files

It is damn simple, like the examples below

You'll first need to define the aws bean that is provided by the plugin:

class MyController {
    def aws
    def myAction = {
        (...)
    }
}

After that, you will use the aws bean and call the s3() method. This will return a helper for S3 Service. After that, you set the target bucket and then call the delete method.

Deleting files stored on the root of some bucket (without path):

To delete the "photo.jpg" file stored under "my-app-bucket-photos" bucket (http://my-app-bucket-photos.s3.amazonaws.com/photo.jpg)

aws.s3().on("my-app-bucket-photos").delete("photo.jpg")

Deleting files stored in some path of one bucket:

To delete the "avatar.jpg" file stored under "my-app-bucket-avatars" bucket and path "/users/lucastex/" (http://my-app-bucket-avatars.s3.amazonaws.com/users/lucastex/avatar.jpg)

aws.s3().on("my-app-bucket-avatars").delete("avatar.jpg", "/users/lucastex/")